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February 13,2004. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates to user interface devices for use with a computer, and in 
particular to computer mice, trackballs, touchpads, multiple-parameter pointing and data 
entry devices, and user interface metaphors. 

2. Description of the Related Art 
User interface devices for data entry and graphical user interface pointing have been 

known for many years. The most common devices include the computer mouse (usually 
attributed to English, Engelbart, and Berman "Display-Selection Techniques for Text 
Manipulation, IEEE Transactions on Human Factors in Electronics, pp. 5-15, vol. HFE-8, 
No. 1, March 1967), the trackball, the touchpad in both finger-operated (for example, the 
various finger-operated devices produced by Symantec Corp., of Springfield, Oregon) and 
stylus-operated (for example, products used with desktop workstation computers - Wacom 
Technology Corp., of Vancouver, Washington) versions, and display-overlay touchscreens. 
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Other historical and exotic devices include various types of light pens and the Data Glove 
(produced by VPL Research, Inc., of Redwood City, California). 

Most user interface devices for data entry and graphical user interface pointing 
commonly used with computers or with equipment providing computer-like user interfaces 
have two wide-range parameter adjustment capabilities that are usually assigned to the task 
of positioning a screen cursor within a two-dimensional display. In many cases, one, two, or 
three binary- valued "discrete-event" controls are provided, typically in the form of spring- 
loaded push-buttons. 

More recently, computer mice have emerged that provide an additional "scroll" 
finger-wheel adjustment (for example, between two control buttons) to provide a third wide- 
range parameter adjustment capability (for example, various products developed by Logitech 
Inc., of Fremont, California). A mouse of this configuration is often referred to as a "Scroll 
Mouse" since this third wide-range parameter is typically assigned the task of positioning a 
vertical scroll bar in an actively selected window. This additional finger-wheel adjustment 
may also operate as a spring-loaded push-button, thus providing an additional binary- valued 
"discrete-event" control. Typically this additional binary-valued "discrete-event" control is 
used to turn on and off an automatic scrolling feature which controls the rate and direction of 
automatic scrolling according to vertical displacement of the displayed cursor. 

SUMMARY OF THE INVENTION 

E In accordanc e with embodiments of the invention include - a freely-rotating 
trackball for simultaneously detecti ng one, two, or three independent directions of its non- 
rotational displacement, and as many as three independent directions (roll, pitch, and yaw) of 
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its rotation. In various implementations, non-rotational displacement of the trackball may be 
measured or interpreted as a widely-varying user interface parameter or as a discrete "click" 
event. Signal processing may be used to derive three independent rotation components (roll, 
pitch, and yaw) from more primitive sensor measurements of the trackball. The invention 
provides for trackball displacement and rotation to be sensed by a variety of sensing 
techniques including optical, magnetic, electromagnetic, capacitive, resistive, acoustic, 
resonance, and polarization sensor. The system may be used to provide an extended number 
of simu ltaneously interactiv e user interface parameters, and may itself be incorporated into 
larger user interface structures, such as a mouse body. 

two indep e nd e ntly adjustable and positionabl e cursors ar e e mploy e d in a visual int e rfac e for 
e diting an electronic document. The document may comprise text s ymbol s , t e xt objects, and 
graphics obj e cts, among oth e rs. If both cur s or s ar e locat e d in clos e proximity within the 
docum e nt, then these cursors may be s imultaneously di s play e d in a singl e window. 
Oth e rwis e , two ar e as of th e docum e nt, each comprising one of th e cursors, may b e 
simultaneously display e d using a separate window for each cursor, or alternat e ly s e l e cted and 
display e d in a s ingl e window. Copy or cut operations may b e mad e with on e cursor, while 
paste operation s may be repeat e d using the second cursor. Cursor locations may b e left 
unchang e d, or th e y may b e moved betwe e n or within e diting op e rations. Th e two cursors 
may b e fr ee ly controlled by an e nhanced pointing device, such a s a mou s e with an added 
touchpad or trackball, or sequ e ntially s e l e ct e d and controlled by a conv e ntional mous e . 

In accordance with embodiments of the invention, a traditional hand-movable 
computer mouse is configured with an additional user interface sensor. For convenience, the 
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term "user interface sensor" will be used herein to collectively refer to devices such as 
trackballs, touchpads, mouse devices, scroll-wheels, joysticks, and other such devices. 

In one aspect of the invention, the addition of a user interface sensor provides 
alternative physical modalities for the same pair of adjustable parameters so that a user may 
switch between using the user interface device as a traditional hand-movable computer 
mouse and using the user interface device as a trackball or touchpad. 

In another aspect of the invention, the addition of a user interface sensor provides 
alternative resolution modalities for the same pair of adjustable parameters so that a user may 
switch between using the invention as a traditional hand-movable computer mouse to obtain 
one level of parameter adjustment resolution, and using the invention as a trackball or 
touchpad, for example, to obtain a different level of parameter adjustment resolution. 

In another aspect of the invention, the addition of a user interface sensor provides 
alternative types of warping modalities for the same pair of adjustable parameters so that a 
user may switch between using the invention as a traditional hand-movable computer mouse 
to obtain one type of parameter adjustment (for example, linear) and using the invention as a 
trackball or touchpad, for example, to obtain a different type of parameter adjustment (for 
example, logarithmic, gamma-corrected, arccosine, exponential, etc.). 

In another aspect of the invention, the addition of a user interface sensor provides 
alternative offset modalities for the same pair of adjustable parameters so that a user may 
switch between using the invention as a traditional hand-movable computer mouse to obtain 
one type of centering of parameter adjustment and using the invention as a trackball or 
touchpad, for example, to obtain a different centering of parameter adjustment. 
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In another aspect of the invention, the addition of a user interface sensor may be used 
to provide additional parameters that may be simultaneously controlled. 

In another aspect of the invention, the addition of a user interface sensor may be used 
to provide additional parameters that are of a different isolated context from those assigned to 
a traditional hand-movable computer mouse. 

In a further more detailed aspect of the invention, the addition of a touchpad may be 
used to provide many additional parameters that are of a different context than those of a 
traditional hand-movable computer mouse. 

In a further more detailed aspect of the invention, the touchpad may be a null-contact 
touchpad adapted to measure at least one maximum spatial span of contact in a given 
direction. 

In a yet further detailed aspect of the invention, the null-contact touchpad is adapted 
to measure at least one maximum spatial span of contact in a given direction at a specifiable 
angle. 

In an additional further detailed aspect of the invention, the null-contact touchpad is 
adapted to measure pressure applied to the null-contact touchpad. 

In a further more detailed aspect of the invention, the touchpad may comprise a 
pressure sensor array touchpad adapted to measure, among other things, one or more of the 
following: the rocking position of a contacting finger in a given direction; the rotational 
position of a contacting finger; the pressure of a contacting finger; and parameters relating to 
a plurality of contacting fingers. 

In another aspect of the invention the addition of a user interface sensor may be 
realized via a replaceable module accepted by an adaptation of a traditional hand-movable 
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computer mouse. In this implementation, a user may initially obtain the invention in one 
configuration and field-modify it to another configuration. 

In another aspect of the invention, a traditional hand-movable computer mouse may 
be implemented as a removable module in a laptop computer or other affiliated equipment, 
and may include a wireless link with the laptop computer or other affiliated equipment. 

In yet a further aspect of the invention, a traditional hand-movable computer mouse is 
implemented as a removable module in a laptop computer or other affiliated equipment, and 
the mouse further comprises a user interface sensor. 

In another aspect of the invention, a traditional hand-movable computer mouse 
additionally comprises a trackball or touchpad, for example. In this aspect, the mouse 
comprises a wireless link to an associated computer or other affiliated equipment. 

In another aspect of the invention, a visual display is provided. 

In another aspect of the invention, an-auditory output is provided. 

In another aspect of the invention, two or more individual user interface sensors 
may be combined without incorporation of such sensors with a traditional hand-movable 
computer mouse. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The above and other aspects, features and advantages of the present invention will 
become more apparent upon consideration of the following description of preferred 
embodiments taken in conjunction with the accompanying drawing figures, wherein: 
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Figures la-li illustrate various exemplary implementations involving merging, 
selecting, multiplexing, and preprocessing distributed in various ways between the body of 
the user interface device and an associated piece of equipment; 

Figures 2a-2c depict an embodiment of the invention comprising a traditional mouse 
fitted with a trackball, illustrating three exemplary button configurations; 

Figures 3a-3c depict an embodiment of the invention comprising a traditional mouse 
fitted with a touchpad, illustrating three exemplary button configurations; 

Figures 4a-4d depict various exemplary degrees of freedom that may be measurably 
assigned to a trackball for interactively controlling parameters in a user interface; 

Figure 4e depicts a freely rotating trackball and associated displacement and rotation 

sensors; 

Figures 5a-5d depict various exemplary degrees of freedom that may be measurably 
assigned to a touchpad for interactively controlling parameters in a user interface; 

Figure 6 depicts an exemplary implementation of the invention directed towards the 
control of both a traditional text cursor and a dual-scrollbar in a typesetting application; 

Figure 7 depicts an exemplary implementation of the invention directed towards the 
active selection from a clip-art or symbol library and adjustment of positioning or other 
attributes of the active selection in a drawing or layout application; 

Figure 8 is a flowchart showing exemplary operations and overhead involved in 
selecting and adjusting a specific pair of parameters from among a larger group of adjustable 
parameters; 
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Figures 9a-9b illustrate how the exemplary operations and overhead depicted in 
Figure 8 introduce excessive overhead in situations where many parameters with a larger 
group of adjustable parameters must be adjusted in pairs; 

Figures 10a- 10b illustrate one technique for adding an additional scroll-wheel to a 
conventional scroll-wheel mouse; 

Figures 1 la-1 lb illustrate a simple example of open adjustments being made within 
various levels of hierarchy of graphical object groupings; 

Figure 12 illustrates aspects of the 3D orientation of an object in 3-dimensional space, 
and in particular the three coordinates of position and the three angles of rotation; 

Figures 13a-13b illustrate one technique for using two cursors in a text cut-and-paste 
operation; and 

Figures 14a-14d illustrate exemplary embodiments of a mouse where the traditional 
mouse buttons have been replaced by trackballs or touchpads. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In the following description, reference is made to the accompanying drawing figures 
which form a part hereof, and which show by way of illustration specific embodiments of the 
invention. It is to be understood by those of ordinary skill in this technological field that 
other embodiments may be utilized, and structural, electrical, as well as procedural changes 
may be made without departing from the scope of the present invention. 

By way of overview, a number of different applications that take advantage of the 
functionality of additional, wide-range adjustment parameters will now be discussed. In one 
example, an additional finger-wheel adjustment device providing a third, wide-range 
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parameter adjustment capability is typically assigned to vertical scroll bar positioning. In 
accordance with the invention, such a design may be supplemented with a fourth, wide-range 
parameter adjustment capability so that a horizontal scroll bar position control may be 
achieved. With the increasing popularity of the web (with many web pages wide enough to 
require horizontal scrolling) and publisher layout tools (typically involving pages wide 
enough to require horizontal scrolling), as well as the need for simultaneous interactive 
horizontal and vertical scrolling actions that do not disturb a screen cursor location when 
using "zoom" controls, a fourth wide-range parameter adjustment capability in traditional 
user interface devices for data entry and graphical user interface pointing is quite valuable. 

There are many other potential uses for additional wide-range adjustment parameters 
in traditional user interface devices for data entry and graphical user interface pointing. 
Some opportunities have wide-range applicability, such as in providing interactive separate 
adjustment of the selections for "cut" or "copy" operations from the interactive adjustment of 
insertion location or selection for a "paste" operation. Other opportunities are more 
specialized but still widely applicable, such as making an active selection from a clip-art or 
symbol library and adjusting the position or other attributes of said active selection in a 
drawing or layout application. Yet other opportunities may be very specialized, such as in 
3D modeling, data visualization, advanced color adjustment, lighting control, machine 
control, or audio and image signal processing. 

There are many opportunities for adjusting the same two widely- varying parameters 
in more than one way. For example, one user interface modality (such as normal mouse 
operation) may be used for normal parameter adjustment, while a second user interface 
modality may be used for adjustments involving a different resolution, warping (i.e., 

2738-36 9 



SUBSTITUTE SPECIFICATION— REDLINED VERSION SHOWING CHANGES 



logarithmic, gamma-corrected, arccosine, exponential, etc.), centering offset, etc. Another 
important case is where the same two widely-varying parameters are controlled with the 
same resolution, warping, offset, etc., but in a different user interface modality (e.g., a 
trackball or touchpad may have some advantages in certain situations over use of a traditional 
mouse). A more widely applicable example is that of responding to and preventing 
hand/wrist/arm fatigue and injury. A traditional mouse fitted with an additional user 
interface sensor allows a user to interchangeably enter information with either the mouse 
body or another user interface sensor, changing which user interface modality is used 
(obtaining the same results with either) to relieve fatigue or pain, or prevent injury. 

More specifically, the addition of a user interface sensor provides many opportunities 
for alternative means of adjustment of a common pair of adjustable parameters. The user 
may benefit from having both adjustment modalities available, changing modalities as 
needed or desired. For example: 

• A user may simply switch between using the invention as a traditional hand- 
movable computer mouse and using the invention as another kind of user 
interface sensor. 

o The user may benefit from having both modalities available to avoid 
or in response to hand fatigue. 

o The user may also benefit from having both modalities available due 
to the type of pointing or data entry interaction needed - depending on 
the case, one type of modality may perform better than another. 

• The trackball, touchpad, or other user interface sensor apparatus may be used 
to provide alternative resolution modalities so that a user may switch between 
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using the invention as a traditional hand-movable computer mouse to obtain 
one level of parameter adjustment resolution and using the invention as a user 
interface sensor to obtain a different level of parameter adjustment resolution. 

• The trackball or touchpad may be used to provide alternative warping 
modalities for the same pair of adjustable parameters so that a user may 
switch between using the invention as a traditional hand-movable computer 
mouse to obtain one type of parameter adjustment (for example, linear) and 
using the invention as another kind of user interface sensor to obtain a 
different type of parameter adjustment resolution (for example, logarithmic, 
gamma-corrected, arccosine, exponential, etc.). 

• The user interface sensor may be used to provide alternative offset modalities 
for a common pair of adjustable parameters so that a user may switch between 
using the invention as a traditional hand-movable computer mouse to obtain 
one centering of parameter adjustment and using the invention as another kind 
of user interface sensor to obtain a different centering of parameter 
adjustment. These modalities can provide one or more "location bookmarks" 
for cursor location, each affiliated with a sub-context within an interactive 
application. 

Further, the addition of another user interface sensor provides many opportunities for 
the simultaneous adjustment of additional parameters that may or may not require 
simultaneous interactive control. The traditional computer mouse may be used to 
simultaneously adjust two parameters while the additional user interface sensor may be 
configured to allow the fingers to simultaneously adjust at least two additional parameters. 
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In some applications, these additional parameters may be closely related to those assigned to 
the traditional computer mouse. For example, the traditional computer mouse may be used to 
simultaneously adjust the location within a window of a text, graphic, or other object, while 
the additional user interface sensor allows the fingers to be used to adjust the type or 
attributes of the text, graphic, or other object. In other applications, these additional 
parameters may be of a different isolated context from those assigned to the traditional 
computer mouse. For example, the traditional computer mouse may be used to 
simultaneously adjust two parameters dealing with affairs within an active application 
window, while the addition of another user interface sensor allows the fingers to be used to 
adjust at least two additional parameters dealing with broader window system affairs such as 
vertical and horizontal scrollbars, window selection, window resizing, etc., or intermediate- 
level affairs such as zoom control, help-window navigation, clip-art selection, etc. Another 
application would be to provide separate adjustment of selections for "cut" or "copy" 
operations from the adjustment of insertion location or selection for a "paste" operation. 

In instances of the invention involving the addition of a touchpad, the touchpad may 
be configured and/or enhanced to allow the fingers to adjust three or more additional 
interactive measured parameters. These additional interactive measured parameters may be 
assigned to control more sophisticated interactive affairs such as 3-dimensional space 
position, 3-dimensional space orientation, color model navigation, image or audio processing 
parameter settings, etc. 

The additional interactive measured parameters (above the two typically associated 
with traditional touchpads) may be provided in a number of ways. For example, the 
touchpad may be a relatively low-cost null-contact touchpad that has been adapted to 
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measure at least one maximum spatial span of contact in a given direction. The user may 
also control an additional parameter by varying the width between the spatial extremes of a 
single point of contact (i.e., how much finger flesh makes contact with the pad) or multiple 
points of contact (i.e., the spread between two contacting fingers). As there are two 
geometrically orthogonal sensing directions on a touchpad, this provides the user with a 
method for controlling four total parameters from a touchpad. Further, rotational 
transformations or other methodologies may be used to measure the angle of rotation of an 
oblong contact profile. The measured angle may be used as a fifth interactive parameter, 
and/or used to adapt the measurement of maximum spatial span of contact in an arbitrary 
angle. The null-contact touchpad may be further adapted to measure pressure applied to the 
null-contact touchpad via, for example, use of an attached pressure sensor. The pressure may 
be used as a sixth interactive parameter, and may or may not have rapid pressure changes 
recognized as 'tap' or 'click' events. 

Another way to provide the additional interactive measured parameters (above the 
two typically associated with traditional touchpads) with a touchpad is to implement the 
touchpad with a pressure sensor array. Through use of operations effectively amounting to 
image processing, a pressure sensor array touchpad can be adapted to measure the rocking 
position of a contacting finger in two orthogonal directions, as well as the rotational position 
and average pressure of a contacting finger. Thus a pressure sensor array touchpad can be 
adapted to provide up to six widely variable interactive adjustable parameters from the 
contact of a single finger. A pressure sensor array touchpad can be further adapted to 
measure parameters relating to a plurality of contacting fingers. 
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All of these considerations and others demonstrate the potential value in providing the 
addition of another user interface sensor to a traditional hand-movable computer mouse. In 
the descriptions to follow, various implementations and exemplary applications of exemplary 
embodiments are considered and explained. 

1. Exemplary Signal Flow and Processing 

The invention provides for a wide range of signal flow and processing configurations 
and implementations. Figures la - li illustrate various exemplary implementations involving 
merging, selecting, multiplexing, and preprocessing distributed in various ways between the 
body of the user interface device and an associated piece of equipment. Figures la - Id 
concern the aggregated pair of user interface sensors in isolation, while Figures le - li 
address arrangements where some functions of the invention are performed in the associated 
external equipment. It is noted that the invention further provides for any of these exemplary 
functionalities, as well as other functionalities, to be combined or made selectable. In any of 
the exemplary implementations disclosed herein, power may be supplied to these 
implementations by the associated external equipment or by other devices such as batteries, 
storage capacitors, photoelectric devices, and the like. 

Figure la shows an implementation 100a featuring two user interface sensors 101, 
102, each of which may be a particular type of user interface sensor (which again may be a 
trackball, touchpad, mouse, or other user interface device) that can be collocated within the 
common physical enclosure 100 (demarcated by the dotted-line boundary). First user 
interface sensor 101 produces signal 108 and second user interface sensor 102 produces 
another signal 109, which are directed to a merge or select function 103. The merge or select 
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function produces outgoing signal 110 which is provided to associated external equipment. 
Here signals 108, 109 (from first and second user interface sensors 101, 102) would typically 
lose their individual identities within the outgoing signal 110 and as such may be used or 
processed interchangeably (without individual attribution to either the first or second user 
interface sensor) by the associated external equipment. 

Merge or select function 103 may take several forms in various implementations. For 
example, in one embodiment it may simply be fixed to only perform a merge operation. In 
another embodiment it may only provide a selection function; here the selection function 
may be controlled by the user using a switch or some sort of action, or the selection may be 
remotely controlled by external equipment. As an alternative, merge or select function 103 
may instead provide a user adjustable "merge" or "select" function. 

Figure lb shows an implementation that is similar to that of Figure la. The primary 
difference is that the Figure lb implementation 100b replaces merge or select function 103 of 
Figure la with multiplex function 104 to produce outgoing signal 110. Here signals 108, 109 
(from first and second user interface sensors 101, 102) retain their individual identities within 
outgoing signal 110 and as such may be used or processed separately by the associated 
external equipment. 

Figure lc shows implementation 100c which is similar in many respects to that of 
Figure la. However, the Figure lc embodiment utilizes preprocessor 105 applied to signal 
109 to produce processed signal 109a. The processed signal 109a, along with signal 108, is 
directed to merge or select function 103, resulting in outgoing signal 110. Preprocessor 105 
may thus introduce a pre-processing step (such as resolution modification, warping 
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modification, offset modification, etc.) on signal 109 to produce a signal of distinguished 
value from that of signal 108. 

Figure Id illustrates another exemplary implementation lOOd which is similar to that 
of Figure lc but with an additional preprocessor 106 applied to signal 108. Preprocessor 106 
produces processed signal 108a, which along with signal 109a, is directed to merge or select 
function 103 to produce outgoing signal 110. Preprocessor 106 may therefore introduce a 
pre-processing step (such as resolution modification, warping modification, offset 
modification, etc.) on signal 108 to produce a signal of either equivalent or distinguished 
value from that of signal 109a. 

Figure le shows implementation 100b of Figure lb (which features two user interface 
sensors 101, 102 and a multiplex function 104 producing an outgoing signal 110) used in 
conjunction with subsequent functions provided by associated external equipment 150a. 
These subsequent functions are shown within the functional boundary 150 of the associated 
external equipment 150a. 

Outgoing signal 110 from the common physical enclosure 100, is presented to 
demultiplexer 117 within external equipment 150a. Demultiplexer 117 produces signal 118 
corresponding to or associated with pre-multiplexed signal 108, and an additional signal 119 
corresponding to or associated with the pre-multiplexed signal 109. Here, signals 118, 119 
are presented to merge or select function 113 producing merged or selected signal 120. This 
implementation is functionally similar or equivalent to that of Figure la except that the 
various types of merge or selection functions 103 are provided within the associated external 
equipment 150a (for example in software, perhaps within an application where it is 
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customized for the needs of that application) rather than being provided within physical unit 
100b. 

Figure If shows implementation 100b of Figure lb used in conjunction with 
subsequent functions provided by associated external equipment 150b, and similar to that of 
Figure le, except signal 119 produced by demultiplexer 117 is directed to preprocessor 115 
to produce processed signal 119a before being sent to merge or selection function 113. This 
implementation is thus functionally similar or equivalent to that of Figure lc except that the 
various types of merge or selection 103 and preprocessor 105 functions in Figure lc are 
provided within the associated external equipment 150b (for example in software, perhaps 
within an application where it is customized for the needs of that application) rather than 
being provided within the physical unit 100b. 

Figure lg shows implementation 100b of Figure lb used in conjunction with 
subsequent functions provided by associated external equipment 150c. This arrangement 
expands on that shown in Figure If in that signal 118 produced by demultiplexer 117 is 
directed to preprocessor 116 to produce processed signal 118a. The processed signal is then 
sent to merge or selection function 113. This implementation is thus functionally similar or 
equivalent to that of Figure Id except that the various types of merge or selection 103 and 
preprocessor 105, 106 functions in Figure Id are provided within the associated external 
equipment 150c (for example in software, perhaps within an application where it is 
customized for the needs of that application) rather than being provided within the physical 
unit 100b. 

Figure lh shows implementation 100b of Figure lb used in conjunction with 
subsequent functions provided by associated external equipment 150d. Here again, as in the 
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arrangement of Figure If, signal 119 produced by demultiplexer 117 is directed to 
preprocessor 115 to produce processed signal 119a. In contrast to other embodiments, 
processed signal 119a is not directed to merge or selection 113 function and retains its 
identity for use by a different destination from that of signal 118 within associated external 
equipment 150d. 

As a final illustrative example in this series, Figure li shows implementation 100b of 
Figure lb used in conjunction with subsequent functions provided by associated external 
equipment 150e. Here, as in the arrangement of Figure Ig, signals 118, 119 produced by 
demultiplexer 117 are directed to preprocessors 115, 116 to produce processed signals 118a, 
119a. In contrast to other embodiments, processed signals 118a, 119a are not directed to 
merge or selection 113 function and thus retain their identity for use by differing destinations 
within associated external equipment 150e. 

Having presented various exemplary signal flows and processing realizations 
provided for by the invention, attention is now directed to exemplary implementations 
utilizing specific types of user interface sensors. It is to be understood that the various 
sensors, techniques and methods disclosed herein may be implemented using computer 
software, hardware, and firmware, and combinations thereof. 

2. Implementations Utilizing Specific Types of Additional User Interface Sensors 

In this section, a number of exemplary implementations of the invention utilizing 
various types of additional user interface sensors added to an original user interface sensor or 
device. The first three sections address cases where the original user interface sensor is a 
movable mouse and the additional user interface sensor is a trackball, touchpad, or other 
exemplary technology, including additional scroll-wheels. Then exemplary adaptations of 
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trackballs and touchpads, each traditionally used to provide simultaneous adjustment of two 
interactive widely- varying parameters, are extended to provide simultaneous adjustment of as 
many as six interactive widely-varying parameters and other forms of control. This section 
continues by presenting exemplary implementations where the original user interface sensor 
is not a mouse, where there are a plurality of additional user interface sensors, where there is 
a visual display or auditory output, where the additional user interface sensor is a removable 
module, and where the implementation itself is a removable module. 

2.1 Trackball Implementations of Additional User Interface 
Sensors 

Figures 2a - 2c illustrate a number of implementations provided for by the invention 
where a trackball controller is used as an additional user interface sensor apparatus 
incorporated into a traditional hand-movable computer mouse. In each of these 
implementations it is understood that the trackball may be freely operated without disturbing 
previous or currently-varying parameter adjustments made by the mouse. 

In one implementation, a trackball controller is added to the top surface of a 
conventional computer mouse as depicted in Figure 2a. The conventional mouse buttons 
may be located in various places in view of the presence of the trackball and in synergy with 
it. In the configuration depicted in Figure 2a, buttons 201 and 202 are located on the surface 
of mouse 200; button 201 being on the left of the trackball and button 202 being on the right 
of the trackball. 

In a second configuration depicted in Figure 2b, buttons 231 and 232 are now 
separated and located on the sides of mouse 230 as is the case with many trackball interfaces; 
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button 231 being on the left side of mouse 230 and button 232 being on the right side of the 
mouse. 

In a third possible configuration depicted in Figure 2c, elongated buttons 261 and 262 
are shown located on the surface of the mouse 260; button 261 wraps around the left of the 
trackball and button 262 wraps around the right of the trackball The elongated buttons 261 
and 262 may be positioned so that a user can readily and rapidly move fingers from trackball 
265 to buttons 261 and 262, or even operate one of these buttons with one finger while 
another finger contacts trackball 265. 

It is noted that unlike the touchpad described below, the trackball has an effectively 
unconfined range of contiguous data entry. 

2.2 Touchpad Implementations of Additional User Interface Sensor 

Figures 3a - 3c illustrate a number of exemplary implementations provided for by the 
invention where a touchpad controller is used as an additional user interface sensor 
incorporated into a traditional hand-movable computer mouse. In each of these 
implementations it is understood that the touchpad may be freely operated without disturbing 
previous or currently-varying parameter adjustments made by the mouse. 

In one implementation, trackball 205 in Figure 2a can be replaced with touchpad 305 
as shown in Figure 3 a. Additionally, this touchpad implementation can also support the 
alternative button configurations of Figures 2b and 2c. By way of illustration, Figure 3b 
shows buttons 331, 332 positioned on either side of the mouse body 330, while Figure 3c 
shows elongated buttons 361, 362 on the surface of the mouse wrapping around either side of 
touchpad 365. 
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It is noted that unlike the trackball, the touchpad typically has a confined maximum 
range of data entry by contiguous operation of a finger, stylus, etc. 

2.3 Other Implementations of Additional User Interface Sensors 

In accordance with other embodiments, the invention provides for still other types of 
additional user interface sensors. In each of these embodiments it is understood that any of 
these user interface sensors may be freely operated without disturbing previous or currently 
varying parameter adjustments made by the mouse or other associated device. 

As a first example, an X-Y joystick may be used in place of the trackball or touchpad 
described above. The joystick may have a spring-return or may retain the last position it was 
placed. Similar to the touchpad and unlike the trackball, the X-Y joystick typically has a 
confined maximum range of travel. 

As another example, two or more scrolling finger wheels may be used in place of the 
trackball or touchpad described above. The scrolling finger wheels may be implemented 
with an unconfined maximum range of travel similar to the trackball, or with a confined 
range of travel like the touchpad and X-Y joystick. In this embodiment, it may be 
advantageous to have one or more finger scroll-wheels mounted with its adjustment direction 
perpendicular to that of another finger scroll-wheel so that each wheel may be appropriately 
associated with vertical and horizontal scroll bars of a window, or other useful orthogonally- 
based user interface metaphor. For example, looking ahead to Figures 10a and 10b, 
embodiments 1000 are depicted comprising the usual components of a scroll-wheel mouse 
(mouse body 1001, buttons 1011 and 1012, and the usual scroll wheel 1021) complemented 
with an additional scroll wheel 1022 with adjustment direction perpendicular to that of finger 
scroll-wheel 1021. 
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As another example, two or more rotating control knobs may be used in place of the 
trackball or touchpad described above. Like the scrolling finger wheels, the rotating control 
knobs may be implemented with an unconfined maximum range of travel like the trackball or 
with a confined range of travel like the touchpad and X-Y joystick. 

The invention also provides for more exotic types of user interface sensor 
technologies - for example, proximity detectors of various types (RF, infrared, etc.), video 
cameras (using any of the techniques provided in U.S. Patent 6,570,078), and other emerging 
and unforeseen technologies - to be used as the additional user interface sensor aggregated in 
the same physical enclosure as the first user interface sensor. 

2.4 Larger numbers of Interactively Widely-Varying Adjustable Parameters 
from the Additional User Interface Sensor 

In accordance with embodiments of the invention, additional user interfaces may be 
used to capture larger numbers (i.e., more than two) of widely-varying adjustable parameters 
from the additional user interface sensor. Some examples are provided here, but many others 
are possible as may be readily understood by one skilled in the art. 

Figures 4a - 4d address the case of the trackball. Figure 4a illustrates a freely- 
rotating trackball 205 and the two principle orthogonal adjustment directions 401, 402 that 
are responsively resolved and measured in traditional trackball user interface devices. 
However, at least two other physical degrees of freedom may be readily exploited, and at 
least six total parameters can be interactively adjusted and measured. Figure 4b shows the 
application of downward pressure 403 on trackball 205. Such pressure 403 may be applied 
without disturbing current values established in orthogonal adjustment directions 401, 402. 
Further, the trackball may be implemented so that downward pressure 403 may be applied 
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while simultaneously adjusting trackball 205 in orthogonal adjustment directions 401, 402, 
particularly if the signal produced by the measurement of downward pressure 403 
incorporates a modest "grace" zone of non-responsiveness for light pressure values. 
Downward pressure impulses may alternatively be sensed and treated as discrete event 
"taps," as commonly used in contemporary touchpad interfaces found in laptop computers, 
for example. 

Figure 4c shows the application of "yaw" rotation 404 (i.e., rotation around the 
vertical axis) of trackball 205. This yaw rotation 404 may be applied without disturbing 
current values established in orthogonal adjustment directions 401, 402 and can be readily 
measured and adjusted as a widely- varying parameter. Further, by grasping trackball 205 (or 
other operational methods), the yaw rotation 404 and traditional orthogonal adjustment 
directions 401, 402 may be independently and simultaneously adjusted. It is also noted that 
in principle up to six widely-varying physical degrees of freedom can be simultaneously 
measured from a properly configured trackball 205 by placing the trackball 205 in a cradle 
that senses not only downward pressure 403 or displacement but also lateral pressure or 
displacement. As shown in Figure 4d, both forward-backward, non-rotational force 405 and 
left-right, non-rotational force 406 may be applied to the trackball in a manner that the values 
of force or displacement in each of these directions 405, 406 can be independently measured. 
Thus, by grasping trackball 205 (or other operational methods), three rotational directions of 
orientation 401, 402, 404 and three non-rotational directions of force or displacement 403, 
405, 406 may be independently and simultaneously adjusted by a user and measured as six 
independent interactively adjustable user interface parameters. These correspond, effectively, 
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to measurable adaptations of the six degrees of freedom of an orientable object in 3- 
dimensional space as found in classical mechanics and aeronautics - that is: 

• "roll" rotation (adapted to 401 ) 

• "pitch" rotation (adapted to 402) 

• "up-down: displacements (adapted to 403) 

• "yaw" rotation (adapted to 404) 

• "forward-backward" displacements (adapted to 405) 

• "left-right" displacements (adapted to 406). 

Most trackball sensing technologies use optically based techniques for sensing the 
two traditional components of rotation ("roll" and "pitch") of the the-trackball. Tfhe 
trackball 205 itself may be configured with an optical pattern on it with spatially varying 
reflectivity for a range of the light spectrum. The pattern may be such that it can spatially 
vary light reflectively in these two traditional components of trackball rotation. 
Alternatively, two spatially varying reflectivity patterns, each active at different ranges of the 
light spectrum or light polarization, may be superimposed or integrated with the trackball. 

A number of approaches may be used to obtain measurements for all three directions 
of rotation. In one completely optical approach, a second or third spatially varying 
reflectivity pattern active at, respectively, a second or third portion of the light spectrum (or 
light polarization if available) may be superimposed or integrated with the patterns employed 
for traditional "roll" and "pitch" rotation sensing, and an additional optical source and sensor 
420 is used to obtain measurement of the added varying reflectivity pattern , as shown in 
Figure 4e . -Depending on the pattern(s) used, sensor signals 421 may be directly usable or 
may require processing of the three primitive signals measured by the sensors to obtain a 
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clean decomposition of the measurement signals into independent "roll," "pitch," and "yaw" 
signals independently responsive to the "roll," "pitch," and "yaw" components of trackball 
rotation. 

As another alternative, t#*e4rackball 205 may include internally, or on its surface, or 
both, materials with spatially varying patterns of magnetic properties, capacitive properties, 
electromagnetic properties, ultrasonic acoustic properties, resonance phenomena of any of 
these properties, polarization phenomena of any of these properties, etc., individually or in 
combination, each of which may be active at specific ranges of polarization, frequencies, etc. 
These may be used together with or in place of optical measurement approaches. Again, 
depending on the pattern(s) used, sensor signals may be directly usable or may require 
processing of the three primitive signals measured by the sensors to obtain a clean 
decomposition of the measurement signals into independent "roll," "pitch," and "yaw" 
signals independently responsive to the "roll," "pitch," and "yaw" components of trackball 
rotation as is clear to one skilled in the art. It is also noted that the third component of 
rotation of the freely-rotating trackball may be interpreted or even measured as a discrete 
"click" event. 

Similarly, a number of approaches may be used to obtain measurements for one, two, 
or three directions of non-rotational trackball displacement. For example, the-trackball 205 
may be secured in a-saddle 415, typically attached in some manner to housing 410 (for 
example, mouse 200, 230, 260), allowing free rotation of the-trackball 205 but causing any 
displacement actions on the trackball to invoke displacements of the-saddle 415 . The saddle 
displacement may be measured with ^displacement sensor 430 generating displacement 
signals 43 1 . Displacement sensor 430 which itself may comprise one or more pressure, 
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optical, resistive, capacitive, magnetic, electromagnetic, continuous-range sensors, switches, 
etc. It is also noted that one or more components of displacement of the freely-rotating 
trackball may be interpreted or even measured as a discrete "click" event. 

Figures 5a - 5d turn now to the case of the touchpad. Figure 5a illustrates touchpad 
305 and the two principle orthogonal data entry directions 501, 502 that are responsively 
resolved and measured in traditional touchpad user interface devices. The touchpad shown in 
Figures 5a -5d, which provides at least four other physical degrees of freedom, may be 
implemented using the techniques presented in U.S. Patent 6,570,078, for example. 

Figure 5b illustrates the use of downward pressure 503 in the context of a touchpad. 
In contemporary touchpad interfaces, such as those found in laptop computers for example, 
such downward pressure 503 is sensed and utilized as discrete event "taps." However, 
downward pressure 503 may also be measured and adjusted as an independent and 
simultaneously interactive widely- varying parameter. Further, as illustrated in Figure 5c, the 
rotational angle 504 of a finger contacting a touchpad with rough-elliptical contact boundary 
can also be measured as a widely-varying parameter. In Figure 5d, both forward-backward 
505 and left-right 506 components of the tilt of a contacting finger can additionally be 
measured as independent and simultaneously interactive widely-varying parameters. 

The sensing of multiple fingers, the application of contact syntaxes and grammars, 
and other user interface control expansions of an adequately configured touchpad may also 
be achieved using, for example, the techniques presented in U.S. Patent 6,570,078. 

The invention also provides for larger numbers (i.e., more than two) of widely- 
varying adjustable parameters from other types of user interface sensor technologies. In the 
case of an X-Y joystick, the joystick may be configured to rotate on its axis, pulled in and 
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out, fitted with a knob or trackball, etc., in a measurable fashion to provide additional and 
simultaneous interactively adjustable parameters. In the cases of finger scroll-wheels and 
rotational knobs, three or more of these devices may be provided. When implementing video 
cameras, known techniques for the extraction of additional parameters may be used. 
Examples of the various types of video extraction techniques that may be used are presented 
in U.S. Patent 6,570,078. 

2.5 Non-Mouse User Interface Sensors 

In one of its most abstract forms, the invention involves the incorporation of two 
conventional user interface sensors (such as a mouse, trackball, touchpad, joystick, etc.) into 
an embodiment where the user may freely use both of the user interface sensors individually 
or simultaneously with the same hand. As such, the invention provides for implementations 
that do not include a mouse as one of the user interface sensors. For example, two or more 
individual user interface sensors can be combined without need of a traditional hand-movable 
computer mouse. Such an implementation may be accomplished by implementing one of the 
possible user interface sensors in place of the traditional hand-movable computer mouse 
where taught in other descriptions of the invention. This configuration may be useful when 
built into a laptop computer, control console, musical instrument, and test instrument, among 
others. 

In one exemplary implementation of a non-mouse embodiment, a trackball and 
touchpad may be arranged in the same physical enclosure so that the front, middle, or back of 
the palm may freely operate a conventional trackball while one or more selected extended or 
arching fingers may simultaneously or alternatively operate a touchpad. In this example, the 
touchpad may be a conventional touchpad providing two widely-varying simultaneously 
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interactive parameters from a single finger contact, or the touchpad may be a more enhanced 
version providing as many as six widely-varying and simultaneously interactive parameters 
from a single finger contact. The touchpad may also be configured to accept multiple points 
of contact, recognize gestures, support syntax and grammatical constructions using, for 
example, the teachings provided by U.S. Patent 6,570,078. 

In another non-mouse implementation, two trackballs may be arranged in the same 
physical enclosure. In one possible arrangement, the two trackballs may be positioned so 
that they lie parallel to the length of the hand, enabling the front, middle, or back of the palm 
to freely operate a first trackball while one or more extended or arching fingers may 
simultaneously or alternatively operate the second trackball. 

In another arrangement, the two trackballs may be positioned so that they lie parallel 
to the width of the hand, so that the fingers and/or thumb on the left side of the hand may 
operate a leftmost trackball while the remaining fingers and/or thumb on the right side of the 
hand may individually or simultaneously operate a rightmost trackball. In each of these 
arrangements, either or both of the trackballs may be a conventional trackball providing two 
widely-varying and simultaneously interactive parameters, or it may be a more enhanced 
trackball providing as many as six widely- varying and simultaneously interactive parameters 
as described earlier. 

In addition to the just-described embodiments, alternative arrangements, such as the 
combination of a palm-operated trackball and a recessed joystick, and others, are also 
provided for by the invention. 

2.6 Use of More than One Additional User Interface Sensor 
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Typically the arrangements of two non-mouse user interface sensors described above 
in Section 2.5 can also be applied to embodiments of the invention where a mouse user 
interface sensor is used. In such embodiments, a mouse user interface sensor is 
supplemented with at least two additional user interface sensors economically arranged so 
that the two additional user interface sensors may be simultaneously or alternatively operated 
by the same hand. If these embodiments are further configured so the mouse body is readily 
moved with adequate precision via the back of the operating hand, then all three user 
interface sensors may be simultaneously or alternatively operated by the same hand in an 
economically advantageous manner. 

Figures 14a- 14d illustrate some exemplary embodiments of the just-described 
features. Figure 14a illustrates a mouse where traditional mouse buttons have been replaced 
by trackballs 1405a, 1405b. These trackballs 1405a, 1405b may accept a downward- 
pressure impulse and as such act as the traditional mouse buttons, However, trackballs 
1405a, 1405b are also adjustable and each readily provides two or more additional widely- 
variable and simultaneously adjustable parameters in addition to the two parameters adjusted 
by moving the mouse body 1400. 

Figure 14b shows a similar arrangement where traditional mouse buttons have been 
replaced by touchpads 1435a, 1435b. If desired, these touchpads may accept a downward- 
pressure impulse and as such act as traditional mouse buttons, but are also adjustable as 
touchpads and as such each readily provides two or more additional widely-variable and 
simultaneously adjustable parameters. As described in Section 2.5, a single hand may be 
positioned to comfortably operate simultaneously or alternatively both trackballs or both 
touchpads. If these embodiments are further configured so the mouse body is readily moved 
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with adequate precision via the back of the operating hand, then either of these embodiments 
readily provides six to twelve widely- variable and simultaneously adjustable parameters. 

Other configurations are of course possible. For example, the configurations of 
Figures 14a and 14b may be blended as depicted in Figure 14c, or in its mirror image. As 
another example, Figure 14d illustrates a more extreme realization comprising a left- 
fingers/thumb trackball 1465a, a right-fingers/thumb trackball 1465b, a palm trackball 
1465c, and a traditional clickable scroll-wheel 1468. Yet another alternative is to replace one 
or more of the trackballs of the Figure 14d embodiment with a touchpad user interface 
sensor. 

2.7 Incorporation of Visual Display and Auditory Output 

If desired, any of the mouse and non-mouse embodiments may further include a 
visual display. The visual display may provide details of adjustable parameter values, 
operation modalities, etc. The visual display may be physically associated with a physical 
enclosure (such as that of a traditional computer mouse), or may be displayed on the 
computer screen or display of other associated equipment. 

Alternatively or additionally, any of the mouse and non-mouse embodiments may 
further provide auditory output. The auditory output may provide details of adjustable 
parameter values, operation modalities, error conditions in usage of the invention, a condition 
relating to elapsed time or other metric of consistent use of a single usage modality, etc. The 
auditory output associated with the invention may be physically associated with a physical 
enclosure (such as that of a traditional computer mouse), or may be produced by speakers or 
headsets affiliated with the computer or other associated equipment. 
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2.8 Provisions for Field Installation or Replacement of Additional User 
Interface Sensor 

The invention also provides for the user interface sensor to be implemented using a 
replaceable module accepted by an adaptation of a traditional computer mouse. In this 
implementation a user may initially obtain the invention in one configuration and field- 
modify it to another configuration. 

2.9 Implementation as a Module Removable from Affiliated Equipment 

The invention also provides for a traditional computer mouse to be implemented as a 
removable module in a laptop computer or other affiliated equipment, and may include a 
wireless link with such devices. In particular, this removable module may further include 
one or more user interface sensors, with these sensors operable as a traditional trackball or 
touchpad when the invention is stowed in the laptop computer or other affiliated equipment 
in such a way that the invention's traditional hand-movable computer mouse modality is 
unmovable and hence unusable. 

3. Exemplary Applications 

Departing now from the range of extreme realization and embodiments of the 
invention, attention is directed towards particular applications of the invention. A number of 
examples of various embodiments of the invention in a wide range of applications will now 
be presented. Many of these applications are viable with only the simplest physical 
embodiments of the invention (for example, those suggested by Figures 2a - 2c and 3a - 3c). 
In the discussion that follows, particular note is directed towards the discussion in Section 3.3 
involving Figures 8 and 9a - 9b. Although the discussion is motivated by a graphical layout 
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application, the principles of the discussion in Section 3.3 involving Figures 8 and 9a - 9b 
are very general, and the discussion illustrates the power of the invention for various 
applications in almost directly quantifiable terms. 

3.1 Wrist/Hand/Arm-Fatigue Relief and Prevention Application 

The danger and damage stemming from extensive continuous or mis-postured mouse 
usage to wrist, hand, and arms are sadly misfortunate and increasingly well recognized. As 
the present invention provides a plurality of different user interface sensors, it is well suited 
for use in responding to and preventing wrist/hand/arm fatigue due to overuse. In one 
exemplary implementation, user interface parameters can be interchangeably adjusted with 
either the movement of the mouse body or the use of an integrated trackball or touchpad with 
identical task results. Thus a user with a tiring hand can change at will the user interface 
sensor employed according to how the hand feels or the nature of a specific task. In addition, 
to prevent fatigue or injury, the user can also switch back and forth between moving the 
mouse body and using the trackball/touchpad either by free choice or by following auditory 
or visual prompting from a time or usage monitor. 

3.2 Double-Scrollbar Application 

Contemporary mice often feature a small rotating wheel between the buttons for use 
in operating the vertical scroll bar of a window without changing the position of the mouse. 
In one particular application of the invention, the left-right sensing capability of the trackball 
or touchpad may be used to add a similar capability for horizontal scroll bars of a window. 

In a trackball implementation, a user can move the vertical bar 611 of Figure 6 up by 
rotating trackball 205 away from him/herself, or one can move the vertical bar 621 down by 
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rotating trackball 205 towards him/herself. By rotating trackball 205 to the left, the user can 
move the horizontal bar 621 left. Similarly, the user can move the horizontal bar 621 right 
by rotating the trackball 205 to the right. 

In a touchpad implementation, a user can move scroll bar 611 up by sliding the finger 
away from her/himself or move scroll bar 611 down by sliding the contacting finger towards 
her/himself; similarly, the user can move the scroll bar 621 left by sliding the finger to the 
left or move the scroll bar 621 right by sliding the contacting finger to the right. 
In another implementation, the vertical and horizontal scroll bars may be adjusted with a 
conventional scroll-wheel mouse that has been fitted with an additional scroll-wheel. Figures 
10a and 10b depict exemplary embodiments 1000 of such an arrangement which comprise 
the usual components of a scroll-wheel mouse including mouse body 1001, buttons 1011 and 
1012, and traditional scroll wheel 1021, with these components further complemented by an 
additional scroll wheel 1022 with adjustment direction perpendicular to that of finger scroll- 
wheel 1021. Figure 10a illustrates an arrangement where the additional scroll-wheel is 
located closer to the user while Figure 10b illustrates an arrangement where the additional 
scroll-wheel is located farther away from the user. In each of these arrangements, the two 
scroll-wheels are shown co-centered with respect to the mouse body; for simultaneous 
adjustment it may be advantageous to locate the additional scroll-wheel 1022 to one side or 
the other of the centered positions shown in Figures 10a and 10b. One approach useful for 
supporting both left-handed and right-handed users, which may provide additional utility, 
would be to provide two off-centered additional scroll-wheels, one on either side of the 
center line of the mouse body 1001 and conventional scroll-wheel 1021. 
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3.3 Traditional 2D Layout, CAD, and Graphics Applications 

In most contemporary 2 -dimensional layout and graphics applications, such as those 
commonly used for viewgraphs, page layout, electronic CAD, etc., numerous mouse 
operations are necessary for each of the many types of object attribute modification, etc. 
Typically, these mouse operations are required because the mouse only allows for the 
interactive adjustment of two widely- varying parameters at a time, and the user must change 
context several times as the parameters adjusted by the mouse are chosen, adjusted, and then 
replaced with another pair of parameters. The present invention is useful in many of these 
circumstances because it allows for more than two parameters to be adjusted at the same 
time. 

Figure 7 shows an example of a session involving the authoring of a viewgraph. The 
viewgraph authoring task showcased in this example includes the creation of a flowchart 
diagram (here depicting a business workflow process) and as such also illustrates related 
needs and attributes of a 2D CAD program involving layout of a diagram (such as a circuit, 
algorithm, etc.) or physical object (such as a PC board, control panel, semiconductor 
photolithography mask, etc.). In that this example further involves drawing, the example 
also illustrates the related needs and attributes of a paint-box or electronic drafting 
application. 

In this broadly representative application, application window 700 is shown 
comprising menu area 700a and drawing area 700b. Within the drawing area, viewgraph 
title 701 and portion 702 of the flowchart to be drawn, comprising thus far a sequence of 
flowgraph objects connected by arrows, have already been entered and rendered. New 
flowgraph objects may be introduced in standard fashion by selecting the type of new object 
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desired from a palette, initially putting an instance of the selected object type in a convenient 
place in the drawing area, adjusting the size, orientation, color, and/or other attributes, and 
putting into final position. Often a number of the last few steps are interactively cycled 
through multiple times before the newly introduced object is adequately drawn and the user 
directs their attention to the next task. In this example, the palette of available objects of a 
specific high-level task is shown as an overlapping stack of three sub-class palettes 703a, 
703b, 703c, each providing a selection of available objects within that sub-class. Here, for 
example, sub-class pallet 703a has been selected (as indicated by the heavy line) from other 
available objects 705, within the sub-class pallet 703a. Upon selection, an initial highly 
adjustable rendering of a specific instance 714 of the selected object 704 appears in a 
convenient location, which may be selected by the user. 

The specific instance 714, rendered in this highly adjustable initial state, is typically 
surrounded by graphical handles 716 which facilitate sizing, positioning within the drawing 
area, and often at least rotational orientation (for example, using the mouse with the ALT key 
simultaneously held down to interactively adjust the angle of rotation of the object should 
that be needed). Traditionally, the cursor controlled by the mouse 717 can be moved within 
object 714 to relocate it within drawing area 700b or can, as shown in Figure 7, be positioned 
atop one of the graphical handles 716 to permit the mouse to adjust the horizontal and 
vertical scale of object 714, i.e., adjust its size and aspect-ratio. In some application 
packages, the latter adjustment is permitted to collapse the object through to 'zero' thickness 
in one of the adjustment dimensions and continue through to re-render the object in mirror 
image, thus additionally providing a form of vertical and horizontal flip by using the size and 
aspect-ratio resizing. 
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As familiar and widely accepted as these sorts of operations are, there is considerable 
overhead involved in such sequences of repeated selecting and adjusting (and in some cases 
additional deselecting) pairs of parameters from a larger collection of parameters. To see 
several aspects of the power of the present invention, these operations are now examined in 
more detail in generalized form. 

Figure 8 is a flowchart showing tasks involved in selecting and adjusting one of a 
plurality of available pairs of adjustable parameters by using a user interface device 
permitting the adjustment of only one pair of parameters at a time. In Figure 8, the task goal 
is simply to adjust a pair of selected parameters 801 from a larger group of adjustable 
parameters. However, since a larger number of parameters are available for adjustment than 
are available at one time with the user interface sensor, the specific pair of parameters must 
first be selected. In most known graphical user interface systems and methods, this typically 
involves first using the user interface device to control the movement of a cursor to a 
selection area of the graphical interface in a first overhead step 811 and then selecting the 
adjustment context (parameter pair) in a second overhead step 821. 

In some situations the selected pair of parameters may be immediately adjusted in 
goal operation 801, but typically the cursor must then at least be moved, in a third overhead 
operation 812, to a location outside of the selection area affiliated with operations 811 and 
821 to another location (such as a drawing or typing area) affiliated with the context 
(parameter pair) that has just been selected for adjustment. In some situations the selected 
pair of parameters may be immediately adjusted in goal operation 801, but typically the 
context must be activated (for example, by clicking in an open portion of a drawing area or 
selecting an existing object) in a fourth overhead operation 822. 
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After the selected pair of parameters are adjusted (for example, by sizing a rectangle, 
etc.) the cycle may then immediately be repeated in some variant form for another pair of 
parameters, but typically the parameters must be deselected (for example, by another click to 
set the final value) in a fifth overhead operation 823 before the cursor may be moved to the 
selection area in another instance of operation 811. In summary, in order to adjust one pair 
of parameters from a larger group of parameters, as many as five overhead operations (as 
many as two cursor movements 811, 812 and as many as three select/deselect clicks 821, 
822, 823) are commonly required. 

Figures 9a - 9b show broader implications of the overhead called out in Figure 8. 
Figure 9a depicts the sequential adjustment of pairs of parameters chosen from a larger group 
of pairs of parameters in a scenario suggestive of no interactive iteration. One pair of 
parameters is adjusted with up to five overhead operations in action 901, then a second pair 
of parameters is adjusted with up to five overhead operations in action 902, then a third pair 
of parameters is adjusted with up to five overhead operations in action 903, and so on. Here 
the overhead slows things down but may not be a significant encumbrance to the broader 
goal of actions 901, 902, 903, etc. 

In contrast, Figure 9b depicts an interactive adjustment of pairs of parameters from a 
larger group of parameters in a scenario suggestive of one where interactive iteration is 
required, as the setting of one pair of parameters is difficult to complete without setting other 
parameters. Here the overhead is likely a significant encumbrance to the higher goal 
involving the pair- wise adjustment actions 901, 902, 903, etc. For example, consider the 
interactive adjustment of six parameters, one pair at a time, through pair-wise adjustment 
actions 901, 902, 903: not only are up to five operations of overhead involved for each of the 
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pair-wise adjustment actions 901, 902, 903, but a considerable extra number of passes must 
be made through these pair- wise adjustment actions 901, 902, 903 due to the fact that the 
adjustment of some parameters depends on or interacts heavily with the values of other 
parameters. The situation gets even more cumbersome should additional pair-wise 
adjustment operations be required. Figure 9b further shows the potential for one or more 
additional adjustment actions 950 which in principle may be iterated as well as and combined 
with pair-wise adjustment actions 901, 902, 903 (as suggested by fully-connected iteration 
paths 921, 922, 923). In contrast to such sequences or iterative graphs of pair-wise 
adjustment actions, the present invention readily offers, for example, four, six, eight or even 
higher numbers of simultaneously adjustable parameters controllable by the same hand 
which, when selected in a context, eliminate the many overhead operations depicted in 
Figures 8 and 9a - 9b, and the many additional iteration steps depicted in Figure 9b. 

Returning now to the generalized graphical layout situation described earlier and 
depicted in Figure 7, the following operations are routinely performed in 2D graphics, layout, 
and CAD applications: 

A. Selection of palette containing object; 

B. Selection of object from palette; 

C. Selection of "layer" object is to be assigned to (common in CAD, but typically not 
used in standard draw and paint packages); 

D. Adjustment of object placement in drawing; 

E. Adjustment of object sizing; 

F. Adjustment of object rotation; 

G. Adjustment of object line thickness(es); 
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H. Adjustment of object line color(s); 

I. Adjustment of object fill color(s); and 
J. Adjustment of object fill pattern(s); 

Of these, operations B, D, and E are almost always utilized, operations A, G, and I are 
frequently utilized, and operations C, F, H, and J are rarely utilized. 

Thus, in one exemplary application of the invention, it may be advantageous to group 
specific collections of operations that are commonly used together (this may be application- 
specific) so that the benefits of having four or more widely-adjustable interactive parameters 
simultaneously available can be applied to speed the execution of basic common operations. 
For example: 

• Employing a four-parameter version of the invention: 

• Operation 1 : Mouse for operation B 

• Operation 2: Mouse for operation D, trackball or touchpad for operation E 

• Employing a six-parameter version of the invention comprising a 4-parameter 
touchpad: 

• Operation 1 : Mouse for operation B, touchpad finger-location for operation 
D, touchpad finger-tilt for operation E. 

• Employing a six-parameter version of the invention comprising a mouse and two 
trackballs or touchpads: 

• Operation 1 : Mouse for operation B, first trackball/touchpad for operation D, 
second trackball/touchpad for operation E. 

Other operations can be later applied in groupings and operations appropriate for the 
application. 
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As a possible alternative to the preceding example, it may be advantageous to assign 
a principal one of the user interface sensors to the sequential adjustment of each of such 
universal (or otherwise principal) operations and reserve the additional user interface sensors 
for rapid "in-context" interactive access to less frequently used operations. For example: 

• Employing a four-parameter version of the invention: 

• Operation 1 : Mouse for operation B, trackball or touchpad for operation A 
and/or operation C; 

• Operation 2: Mouse for operation D, trackball or touchpad for operation E 
and/or operation F; 

• Operation 3: Mouse for operation G, trackball or touchpad for operation H; 
and 

• Operation 4: Mouse for operation I, trackball or touchpad for operation J. 

• Employing a six-parameter version of the invention comprising a 4-parameter 
touchpad: 

• Operation 1 : Mouse for operation B, touchpad finger-location for operation A 
and touchpad finger-tilt for operation C; 

• Operation 2: Mouse for operation D, touchpad finger-location for operation E 
and touchpad finger-tilt for operation F; 

• Operation 3: Mouse for operation G, touchpad finger-location (and touchpad 
finger-tilt as useful) for operation H; and 

• Operation 4: Mouse for operation I, touchpad finger-location (and touchpad 
finger-tilt as useful) for operation J. 
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• Employing a six-parameter version of the invention comprising a mouse and two 
trackballs or touchpads: 

• Operation 1 : Mouse for operation B, first trackball/touchpad for operation A 
and second trackball/touchpad for operation C; 

• Operation 2: Mouse for operation D, first trackball/touchpad for operation E 
and second trackball/touchpad for operation F; 

• Operation 3: Mouse for operation G, first trackball/touchpad (and second 
trackball/touchpad as useful) for operation H; and 

• Operation 4: Mouse for operation I, first trackball/touchpad (and second 
trackball/touchpad as useful) for operation J. 

As another alternative example, the user may freely assign user interface sensor 
parameters to operations A through J (and others as may be useful) for each of a number of 
steps as may match the task or tasks at hand. These assignments may be stored for later 
retrieval and use, and may be named by the user. The stored assignments may be saved 
along with specific files, specific applications, or as a general template the user may apply to 
a number of applications. It is noted that such variable assignments may be particularly 
useful to users as their hands fatigue, to prevent fatigue or injury, or as an adjustment for a 
temporary or permanent disability. 

3.4 Multi-Resolution Mouse Application 

In another exemplary family of applications, one user interface sensor (for example, 
the mouse body) is used for course adjustment or fine adjustment of user interface 
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parameters, while the additional user interface sensor (for example, a trackball or touchpad) 
is used for the remaining level of parameter adjustment resolution. 

In most user-interface applications it is advantageous to have multiple scales of 
graphical user interface pointing and data entry. Many window systems provide an 
'acceleration' setting which changes the pointing and data entry values on a more significant 
scale used for user interface changes made less frequently. Many applications further 
internally adjust the resolution as the corresponding visual display is "zoomed" in and out. 

In many user interface applications, additional levels of resolution selection may be 
useful. For example, in pointing usage in text work, multiple resolutions would be 
advantageous in amending fine print or in making isolated changes in thumbnail overviews 
of 40% actual size or less. Similarly, in graphics work, fine resolution may be especially 
useful in making fine adjustments to figures. In the fine adjustment of figures, it may be 
further advantageous to employ each of the separate user interface sensors in conjunction 
with corresponding snap-grids of differing grid spacing, particularly if one of the grid 
spacings is a sub-multiple of the other. A potentially useful extension of this would be to 
impose locally-applicable grid spacing on individual graphic or other objects, particularly 
objects which have been resized and hence for which the standard snap-grid spacing is no 
longer useful. 

In a further application, the user interface may be directed towards non-positional 
adjustments, such as the adjustment of a rotation angle or of the color of a graphic object; 
here multiple resolutions may be valuable to make careful adjustments and coarse 
adjustments as needed. Similarly, scroll bars for long documents may also benefit from rapid 
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access to multiple resolution scales, for example one user interface sensor may be used to 
navigate within a page (using a fine-grained navigation scale) while a second user interface 
sensor may be used to navigate across pages (using a coarser-grained navigation scale). 

3.5 Provision of Both Absolute and Relative Positioning 

As discussed earlier, some types of user interface sensors, such as the touchpad and 
X-Y joystick for example, naturally have a limited maximum range of operation while others 
such as a mouse, trackball, and scroll-wheel have an essentially unlimited maximum range of 
operation. Although most user interface sensors are interpreted in relative terms (that is, the 
stimulus from the sensor is interpreted as a command to move a cursor, scroll bar, etc., 
incrementally in some direction relative to a current position), stimulus signals from any of 
these types of user interface signals may be interpreted in either a relative or absolute manner 
with varying degrees of naturalness or problematic qualities. 

The present invention provides for one user interface sensor to be used for absolute 
positioning of a cursor, scroll bar, etc., or other means of parameter adjustment while another 
user interface sensor is used for traditional relative adjustment of such parameters. For 
example, a scroll bar may be adjusted in the usual fashion by a mouse body or trackball and 
in an absolute manner by a touchpad wherein the extreme values of the adjusted parameter 
correspond to the extreme positions at the edges of the touchpad. In one embodiment or 
application setting these two user interface sensors may control the same parameters - here it 
is often the result that the two sensors adjust the same parameters with different resolutions. 
Further, in this situation it is fairly likely that at least one of the resolution scale factors will 
be adjusted automatically. For example, in a document editor, as the number of pages of the 
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document varies, the resolution of the absolute positioning sensor will correspondingly vary 
(so that the extremities in range of, for example, a touchpad correspond to the top of the first 
page and end of the last page) while the relative positioning sensor may retain the same 
incrementing/decrementing vertical scrolling resolution scale regardless of the number of 
pages. 

3.6 Color-Selection Application 

In color adjustment, three parameters are involved in the full interactive span of any 
complete color space (RGB, HSB, YUV, etc.). By adding additional parameters to the 
overall user interface, all three parameters can be adjusted simultaneously rather than simply 
two at a time. As the present invention provides at least four simultaneous interactively 
adjustable parameters overall, it is thus potentially useful for fully interactive color 
adjustment within a complete color space model. Further, should the additional user 
interface sensor be such that it alone provides three simultaneously interactively adjustable 
parameters, the first user interface sensor (for example, the mouse body) may be used as a 
pointer to select objects and the additional user interface sensor may be used to adjust 
attributes of the selected object such as its color, border color, etc. 

3.7 Multi-level Graphic Object Grouping and Editing Application 

In many drawing applications, lower-level graphical or other objects (such as lines, 
basic shapes, and text areas) may be grouped to form an aggregated object. This aggregated 
or "grouped" object (collectively referred to herein as an "aggregated object") can then be 
moved, rotated, flipped, resized, etc. as if it were a lower-level graphic or other object. 
Grouping can also typically be done hierarchically and in mixed hierarchies, i.e., a plurality 
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of lower-level graphical or other objects may first be grouped, and the resulting aggregated 
object may then itself be grouped with other aggregated objects and/or lower-level graphical 
or other objects. 

Often one or more of the lower-level graphical or other objects comprising the 
aggregated object may need modification. In the case of text, most applications permit 
modifications to be made to individual text objects within an aggregated object. However, for 
any isolated adjustment to any other lower-level graphical or other object the aggregated 
object must be first disaggregated or "ungrouped" to completely free the involved lower- 
level graphical or other object from any grouping it had been involved in. After the 
modification, the grouping must be reconstructed. Often this becomes a cumbersome 
situation, particularly where the adjustments within the group are themselves an interactive 
response to other adjustments made within a drawing. 

The additional number of widely-adjustable simultaneously interactive parameters 
made possible by the invention may be advantageously applied to this problem. For example, 
one user interface sensor may be used to navigate the levels of grouping and another user 
interface sensor may be used to perform operations on objects (lower level or "grouped") 
within that level of grouping of the overall aggregated object. 

As an illustrative example, Figure 11a shows a portion 1100 of a larger drawing, the 
portion 1100 featuring box 1101, two arrowed lines 1111, 1112, and grouped object 1102 
(here itself comprising two triangles connected by a line). In this example it is given that 
grouped object 1102 is itself grouped with box 1101 to form a second grouped object, and 
this second grouped object is itself grouped with the two arrowed lines 1111, 1112 to form a 
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third grouped object. The user's task is to modify Figure 1 la so that it becomes Figure 1 lb. 
To do this, effectively the user must, in some order of operation: 

• Reposition grouped object 1102 from its original position in Figure 1 la to a new 
position in Figure lib; 

• Copy or otherwise reproduce grouped object 1 102 to create an accompanying 
grouped object 1102a, and position it within box 1101; 

• Introduce a vertically distributed ellipsis 1103 and position it within box 1101— 
typically, a vertically distributed ellipsis 1103 is either rotated text or itself a fourth 
grouped object created from three aligned text elements; and 

• Ensure elements 1102, 1102a, and 1103 are in the end grouped with box 1101 to form 
the second grouped object, and this second grouped object is itself grouped with the 
two arrowed lines 1111, 1112 to form a third grouped object. 

Utilizing the invention, one user interface sensor is used to select the second group level, and 
the second user interface sensor is used to perform insert, copy, paste, and position 
operations within this level of grouping without any form or type of ungrouping operation. If 
the vertically distributed ellipsis 1103 itself is realized as a fourth grouped object created 
from three aligned text elements, when it is pasted into the drawing via this modality its 1103 
grouping is subordinated appropriately (i.e., structured as a peer to grouped objects 1102, 
1102a) within the second grouping level. 

Although readily implemented using the novel user interface sensors described herein 
that make it particularly easy to simultaneously adjust a plurality of pairs of parameters, the 
aspects of the invention illustrated here can also be implemented with a conventional user 
interface sensor such as a traditional mouse, trackball, or touchpad. In this case, the 
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conventional user interface sensor such as a traditional mouse, trackball, or touchpad must 
first be used to select the level of grouping and then be used to make the desired 
modifications within that level of grouping; to make modifications at a different level of 
grouping, the new level of grouping must be selected in a separate operation, thus adding 
overhead, as depicted in Figures 8 and 9a - 9b. Although this novel ability to move and 
modify arbitrary graphic or other objects within groupings may be implemented in this way, 
having an additional number of widely-adjustable simultaneously interactive parameters - 
made possible by the main themes of the present invention - is clearly more efficient, as 
many or all of the overhead operations depicted in Figures 8 and 9a - 9b can be eliminated 
via usage of the additional widely-adjustable simultaneously interactive parameters. 

3,8 3D Graphic Object Placement and Orientation Application 

CAD and drawing packages that enable 3D object placement and orientation within a 
3D space typically extend the capabilities of traditional 2D layout, CAD, and graphics 
applications as described in Section 3.3 to serve additional geometric needs. As illustrated in 
Figure 12, the placement and orientation of 3D object 1200 within a 3D space 1250 (oriented 
with respect to a reference point 1251) requires that one additional position dimension and 
two additional orientation angles be specified to complete the full collection of three position 
dimensions 1201, 1202, 1203 and three orientation angles 1211, 1212, 1213. 

To interactively adjust these parameters pairwise (or individually with a knob box as 
has been done historically in some systems) involves complex repetitive passes among high- 
overhead operations as depicted in Figures 9a - 9b, for example among steps 901, 902, 903. 
The necessity of making many high-overhead operations, for example moving among steps 
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901, 902, 903, can be functionally disruptive as well as slow and inefficient. The ability to 
interactively freely adjust the full collection of three position dimensions 1201, 1202, 1203 
and three orientation angles 1211, 1212, 1213 is thus of extremely high value. 

The invention provides for a wide range of mappings between the six position and 
orientation parameters 1201, 1202, 1203, 1211, 1212, 1213 involved in the placement and 
orientation of 3D object 1200 within a 3D space, and the large numbers of widely-adjustable 
and simultaneously interactive parameters facilitated by various realizations of the invention. 
As one example, a mouse fitted with two trackballs as in Figure 14a may be used to specify 
these six parameters in various ways. One technique is to use the position of mouse body 
1400 to control two of the position coordinates (for example 1202, 1203), one of the 
trackballs (for example 1405a) to control the orientation angles (1212, 1213) corresponding 
to these two axes, and the remaining trackball (1405b) to control the remaining axis (1201) 
and its corresponding orientation angle (1211). In this example, trackballs 1405a, 1405b are 
configured or used in 2-parameter modalities. 

In another example, the mouse of Figure 14a is fitted with two trackballs, the 
trackballs may be configured in 3-parameter modalities with one of the trackballs used for 
controlling three position dimensions 1201, 1202, 1203 and the second trackball configured 
to correspondingly control the three orientation angles 1211, 1212, 1213. Here the position 
of mouse body 1400 may be used to control other aspects of drawing operations. 

In another implementation, a touchpad configured for 4-parameter operation 
involving two parameters of finger position and two parameters of finger tilt may be 
combined with a trackball configured for 2-parameter operation. In this example, finger 
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position may be used to control two of the position coordinates (for example 1202, 1203), 
finger tilt may be used to control the orientation angles (1212, 1213) corresponding to these 
two axes, and the trackball to control the remaining axis (1201) and its corresponding 
orientation angle (1211). If the configuration includes a mouse body, its position may be 
used to control other aspects of drawing operations. 

In another example, a configuration like that of Figure 14d may use left- 
fingers/thumb trackball 1465a to control a first position coordinate 1201 and its 
corresponding orientation angle 1211, the right-fingers/thumb trackball 1465b to control a 
second position coordinate 1202 and its corresponding orientation angle 1212, and palm 
trackball 1465c to control third position coordinate 1203 and its corresponding orientation 
angle 1213. 

The invention further provides for a wide range of additional mappings and geometric 
metaphors between the user interface sensor geometry and the three position dimensions 
1201, 1202, 1203 and three orientation angles 1211, 1212, 1213 of a 3D object. 

3.9 Multiple Cursors and Cut and Paste Application 

The invention additionally provides for a plurality of pairs of user interface sensor 
parameters to be used to control the respective positions of a corresponding plurality of 
individual cursors, selections, and/or insertion points. Multiple cursors and associated 
operations of multiple selection and insertion points can have many applications. Below, a 
few of these possibilities that would be apparent to one skilled in the art are showcased in a 
cut-and-paste editing example. 
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Cut, copy, and paste operations using traditional user interface devices usually 
involve multiple operations to switch between contexts introducing considerable overhead as 
depicted in Figures 9a - 9b and 8. For instance, Figure 13a illustrates a text editing example 
with text display window 1300 involving the selection of a clause 1320 (highlighted in this 
example) with the intention of relocating it to a new position 1351. In such an operation with 
a traditional 2 -parameter mouse/trackball/touchpad user interface, the cursor is first used to 
select clause 1320 and then used to select the insertion position 1351. 

When writing or editing it is often the case that material needs to be fetched from 
elsewhere and put in the spot where one is currently writing. Here, the cursor is initially in 
the spot where the insertion is to occur and the user must then lose the cursor position 
currently set in this spot to go searching and then to select and cut or copy the material to be 
pasted; following this the user must then search again, perhaps taking considerable time, for 
said initial spot and re-establish the cursor location there. Equally often there are other 
situations where material must be split up and distributed in a number of far-flung places. 
Here, the cursor is initially in the spot where the material to be sequentially divided and 
relocated is originally aggregated; the user must repeatedly select the portion of the 
remaining aggregated material to be relocated and then lose that cursor position to go 
searching for the new destination insertion spot, perform the insertion, and then search again, 
perhaps taking considerable time, for the initial spot and re-establish the cursor location 
there. In both of these cases it would be advantageous if the user could "bookmark" an initial 
cursor location, search and perform the desired fetch or relocation operations, and readily 
return without search to the "bookmarked" location. Although this novel and advantageously 
valuable capability could be realized with a conventional mouse through context redirection 
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operations involving the steps depicted in Figures 8 and 9a - 9b, the present invention 
provides for a wide range of readily realized and easy-to-use implementations. 

The invention may be used in a minimal configuration capable of interactively 
specifying at least two pairs of widely adjustable interactive parameters. Returning to the 
specific example associated with Figure 13a, one pair of parameters is used to set the location 
of first cursor 1301 which is used in a selection operation, while the second pair of 
parameters is used to set the location of second cursor 1351 which is to be used to 
independently set an insertion point. The user may then perform the cut and paste operation 
with a single mouse click, resulting in the outcome depicted in Figure 13b. The relocated text 
clause 1320 has been transferred to a position determined by the insertion cursor 1351 (here 
shown to the left of the cursor 1351; it could just as easily be to the right of it), and cursor 
1301 used to make the selection remains in position. Either cursor 1301 or 1351 may now be 
moved and/or used for other cut, copy, paste, or (via the keyboard) new text insertion 
operations. 

Although in this example the two cursor locations were close enough to be displayed 
in the same window 1300, the value of this application of the invention is significantly 
increased should the two positions be separated by many pages, many tens of pages, or even 
many hundreds of pages of text. Such situations may be handled by any number of 
approaches as is clear to one skilled in the art. In one approach involving a single display 
window, the area comprising the cursor whose corresponding user interface sensor was last 
manipulated is displayed in the single display window. In another approach involving a 
single display window, a click event or other user interface stimulus may be used to toggle 
among the areas comprising the various cursor locations. In yet another approach, at least 
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two windows may be rendered, with one of the cursors displayed and operable within one 
window and a second cursor displayed and operable in a second window. 

The invention also provides for these general principles to be applied to other types of 
objects and applications, such as spreadsheet cells (involving data, formula objects, and cell 
formats), graphical objects, electronic CAD diagrams (where objects may be connected with 
formulas, dynamic models, etc.), and others as will be apparent to one skilled in the art. 

3.10 Simulation, Processing, and Analysis Applications 

Simulation, processing, and analysis applications typically involve a large number of 
parameters which are adjusted to model, affect or investigate the resulting behaviors, end 
results, and/or implications. Conventional 2-parameter user interface devices such as a 
mouse/trackball/touchpad require the user to adjust these parameters pairwise (or 
individually with a knob box as has been done historically in some systems) involving 
complex repetitive passes among high-overhead operations as depicted in Figures 9a - 9b, 
for example among steps 901, 902, 903. As in the case of 3D object positioning and 
orientation, the division among high-overhead operations, for example moving among steps 
901, 902, 903, can be functionally disruptive as well as slow and inefficient. The ability to 
interactively and freely adjust larger collections of parameters simultaneously is thus also of 
extremely high value. 

3.11 Live Signal Processing and Lighting Applications 

In artistic performance, composition, and recording applications, control of large 
numbers of parameters requiring simultaneous interactive adjustment is common. 
Conventional recording, mixing, video, and light control consoles typically have large 
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numbers of controls with carefully designed spatial layouts to facilitate the rapid and precise 
adjustment of multiple parameters via knobs, sliders, pushbuttons, toggle switches, etc. The 
introduction of computer GUIs has added considerable value and new capabilities, including 
"soft" reconfigurable consoles and functional assignments, but in the bargain typically 
encumber users - accustomed to rapid and precise operation of multiple parameters - with a 
2-parameter mouse/trackball/touchpad having the overhead of iterative context-switching 
operations depicted in Figures 8 and 9a - 9b. As in the case of 3D object positioning and 
orientation, the division among high-overhead operations, for example moving among steps 
901, 902, 903, can be functionally disruptive as well as slow and inefficient. The ability to 
interactively freely adjust larger collections of parameters simultaneously is thus also of 
extremely high value. 

3.12 Real-Time Machine Control and Plant Operations 

Similarly, real-time machine control and plant (manufacturing, chemical, energy, etc.) 
operations also traditionally involve controlling a significant number of parameters requiring 
simultaneous interactive adjustment. Conventional real-time machine control and plant 
operation consoles typically have large numbers of controls with carefully designed spatial 
layouts to facilitate the rapid and precise adjustment of multiple parameters via knobs, 
sliders, pushbuttons, toggle switches, etc. The introduction of computer GUIs can add 
considerable value and new capabilities, including "soft" reconfigurable consoles and 
functional assignments, but in the bargain typically significantly encumber users - 
accustomed to rapid and precise operation of multiple parameters - with a 2-parameter 
mouse/trackball/touchpad having the overhead of iterative context-switching operations 



2738-36 



53 



SUBSTITUTE SPECIFICATION— REDLINED VERSION SHOWING CHANGES 



depicted in Figures 8 and 9a - 9b. As in the case of 3D object positioning/orientation and 
artistic applications described earlier, the division among high-overhead operations, for 
example moving among steps 901, 902, 903, can be functionally disruptive as well as slow 
and inefficient. The ability to interactively freely adjust larger collections of parameters 
simultaneously is thus also of extremely high value. 

A very few examples of this category of application where the invention may be 
useful include many forms of robotics control, computer-control manufacturing tools, 
industrial optical and electron microscopy, camera control (pan, tilt, zoom, focus, and/or iris), 
plant process elements (heaters, pumps, values, stirrers, aerators, actuators, activators, etc.), 
and a wide range of other related and divergent possibilities apparent to those skilled in the 
art. 

4. Concluding Remarks 

The present invention at its core provides for a wide range of systems and methods 
for realizing and applying user interfaces providing, in many cases, at least four widely- 
variable simultaneously interactively adjustable parameters. In so doing, the inventiqnmore 
broadly encompasses novel user interface structures, metaphors, and applications readily 
suggested and enabled by the core of the invention but which may be indeed realized in ways 
not involving the core of the invention. 

While the invention has been described in detail with reference to disclosed 
embodiments, various modifications within the scope of the invention will be apparent to 
those of ordinary skill in this technological field. It is to be appreciated that features 
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described with respect to one embodiment typically may be applied to other embodiments. 
Therefore, the invention properly is to be construed with reference to the claims. 



Claims 

What is claim e d is: 

h A m e thod for facilitating comput e r e diting of an e l e ctronic docum e nt, said 

2 e l e ctronic docum e nt comprising a plurality of obj e cts that ar e graphically r e nd e r e d in a 

layout comprising a plurality of uniqu e locations, said m e thod comprising: 
4 positioning a first cursor at a first location within said e l e ctronic docum e nt; 

positioning a s e cond cursor at a s e cond location within said e l e ctronic docum e nt, 

6 wh e r e in said first and second cursors ar e ind e p e nd e ntly displayabl e and ind e p e nd e ntly 

positionabl e at any of said plurality of uniqu e locations of said e l e ctronic docum e nt; 
8 d e fining a s e l e ction string bas e d upon a s e l e ct e d contiguous arrang e m e nt of at l e ast 

on e obj e ct of said plurality of obj e cts; and 
10 transf e rring said s e l e ction string to a location within said electronic docum e nt as 

d e t e rmined by said s e cond location of said s e cond cursor. 

2-. Th e m e thod according to claim 1, wh e r e in said transf e rring compris e s a copy 

2 and past e op e ration. 

2>~. Th e m e thod according to claim 1, wh e r e in said transf e rring compris e s a cut - 

2 and past e op e ration. 



2 



4: Th e m e thod according to claim 1, wh e r e in said transf e rring is r e p e at e d 

r e sponsiv e to us e r input. 
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Th e m e thod according to claim 1, said m e thod furth e r comprising: 

2 positioning said second cursor at a third location within said e l e ctronic docum e nt; and 

transf e rring said s e l e ction string to a location within said e l e ctronic docum e nt as 

4 d e t e rmin e d by said third location of said s e cond cursor 

Th e m e thod according to claim 1, said m e thod furth e r comprising: 

2 r e p e at e dly r e positioning said s e cond cursor at n e w locations within said el e ctronic 

docum e nt; and 

4 transf e rring said s e l e ction string to e ach of a plurality of n e w locations within said 

e l e ctronic docum e nt as d e t e rmin e d by said r e p e at e dly r e position e d s e cond cursor. 

Th e m e thod according to claim 1, wh e r e in said s e lection string is d e fin e d 

2 b e for e said s e cond cursor is position e d at said s e cond location within said e lectronic 
docum e nt. 

& Th e m e thod according to claim 1 , wh e r e in said s e l e ction string is d e fin e d after 

2 said s e cond cursor is position e d at said s e cond location within said e l e ctronic document. 

9-. Th e m e thod according to claim 1 , wh e r e in said first and s e cond cursors are 

2 simultan e ously display e d and positionabl e in a common graphical user int e rfac e window. 

1-& Th e m e thod according to claim 1, wh e r e in said first and second locations are 

2 s e parated by a distanc e such that said first and s e cond cursors are unabl e to bo 

simultan e ously display e d in a common graphical us e r int e rfac e window, and wh e r e in only 
4 on e of said first and s e cond cursors is display e d wh e n said graphical us e r int e rfac e window 

displays a portion of said e l e ctronic docum e nt that compris e s a s e l e ct e d one of said first and 
6 s e cond cursors. 



2 



J-k Th e m e thod according to claim 1, wh e r e in said first and s e cond locations ar e 

s e parated by a distanc e such that said first and s e cond cursors ar e unabl e to bo 
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simultan e ously displayed in a common graphical us e r int e rfac e window, and wheroin only 
4 on e of said first and s e cond cursors ar e positionablo whon said graphical us e r interface 

window displays a portion of said e l e ctronic docum e nt that compris e s a s e l e ct e d one of said 
6 first and s e cond cursors. 

12-. Th e m e thod according to claim 10, wh e r e in said graphical us e r interface 

2 window displays a portion of said el e ctronic docum e nt that compris e s a last positioned on e of 
said first and s e cond cursor s . 

Th e m e thod according to claim 11, wh e r e in said graphical user int e rfac e 

2 window displays a portion of said e l e ctronic docum e nt that compris e s a last position e d on e of 
said first and s e cond cursors. 

\A-. Th e m e thod according to claim 10, wh e r e in said positioning of said first 

2 cursor is controll e d by a first us e r int e rfac e s e nsor, and said positioning of said s e cond cursor 

is controll e d by a s e cond us e r int e rface s e nsor; and wh e r e in 
4 said graphical us e r int e rfac e window displays a portion of said e l e ctronic docum e nt 

comprising on e of said first and s e cond cursors that is associat e d with a last manipulat e d on e 
6 of said first and s e cond us e r int e rfac e s e nsors. 

Th e m e thod according to claim 11, wh e r e in said positioning of said first 

2 cursor is controll e d by a first us e r int e rfac e s e nsor, and said positioning of said s e cond cursor 

is controll e d by a s e cond us e r int e rfac e s e nsor; and wherein 
4 said graphical us e r int e rfac e window displays a portion of said e l e ctronic docum e nt 

comprising on e of said first and s e cond cursors that is associated with a last manipulat e d one 
6 of said first and s e cond us e r int e rfac e s e nsors. 

Wh Th e m e thod according to claim 10, wh e r e in a user int e rfac e s e l e ction e v e nt 

2 s e l e ctiv e ly causes said graphical us e r int e rfac e window to display e ith e r a portion of said 

e l e ctronic docum e nt that compris e s said first cursor, or a portion of said e l e ctronic document 
4 that compris e s said s e cond cursor. 
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VJ-. Th e m e thod according to claim 1 1 , wh e r e in a us e r int e rfac e s e lection e v e nt 

s e l e ctively caus e s said graphical us e r int e rfac e window to display e ith e r a portion of said 
e l e ctronic document that compris e s said first cursor, or a portion of said e l e ctronic document 
that compris e s said s e cond cursor. 

\9n Th e m e thod according to claim 1 , wh e r e in said first cursor is display e d within 

a first graphical us e r int e rfac e window, and said s e cond cursor is display e d within a s e cond 
graphical us e r int e rfac e window. 

19-. Th e m e thod according to claim 1, wh e r e in said first cursor is positionablo 

within a first graphical us e r interfac e window, and said s e cond cursor is positionable within a 
s e parat e , second graphical us e r int e rfac e window. 

2& Th e m e thod according to claim 1, wh e r e in on e or mor e of said plurality of 

obj e cts compris e a t e xt symbol. 

2-k Th e m e thod according to claim 1, wh e r e in on e or mor e of said plurality of 

objects comprise a t e xt obj e ct. 

22r. Th e m e thod according to claim 1, wh e r e in on e or mor e of said plurality of 

obj e cts comprise a graphics obj e ct. 

2& Th e m e thod according to claim 1, said method furth e r comprising: 

g e n e rating int e ractiv e param e ters r e sponsiv e to op e ration of a hand operat e d user 

int e rfac e d e vic e ; and 

s e l e ctiv e ly controlling said positioning of only on e of said first and s e cond cursors at 

any giv e n tim e bas e d upon said g e n e rat e d int e ractiv e param e t e rs. 
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24-. Th e m e thod according to claim 1, said method furth e r comprising: 

g e n e rating at l e ast two diff e rent, int e ractive parameters r e sponsiv e to op e ration of a 

hand operat e d us e r interface d e vic e ; and 

controlling said positioning of said first and s e cond cursors bas e d upon said generated 

at l e ast two int e ractive param e t e rs. 

2$-. Th e m e thod according to claim 2 4 , wh e r e in a first on e of said int e ractive 

param e t e rs is g e n e rat e d by a s e nsor hous e d within said us e r interface d e vic e , wh e r e in said 
s e nsor is adapt e d to d e t e ct mov e m e nt of said u se r int e rfac e d e vic e r e lativ e to two axes 
orthogonal to e ach oth e r, and wh e r e in 

a s e cond on e of said int e ractiv e param e t e rs is g e n e rat e d by an additional user 

int e rfac e s e nsor coupl e d to said us e r int e rfac e d e vice. 

2&. Th e m e thod according to claim 25, wh e r e in said additional us e r interface 

s e nsor compris e s a trackball. 

¥h Th e m e thod according to claim 25, wh e r e in said additional us e r int e rfac e 

s e nsor compris e s a touchpad. 

2& Th e m e thod according to claim 1, said m e thod furth e r comprising: 

g e n e rating at l e ast four ind e p e nd e ntly adjustable int e ractiv e param e t e rs r e sponsiv e to 

op e ration of a touchpad; and 

controlling said positioning of said first and s e cond cursors bas e d upon at l e ast two of 

th e g e n e rat e d at least four int e ractiv e param e t e rs. 
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29-. Tho method according to claim 1 , said m e thod furth e r comprising: 

2 g e n e rating at least two ind e pend e ntly adjustabl e interactiv e param e t e rs r e sponsive to 

op e ration of a first trackball; 
4 g e n e rating at least two independ e ntly adjustabl e int e ractiv e parameters r e sponsive to 

op e ration of a second trackball; and 
6 controlling said positioning of said first and s e cond cursors bas e d upon int e ractiv e 

param e t e rs g e n e rated by on e of said first and s e cond trackballs. 

3-Q: The m e thod according to claim 1, said m e thod furth e r comprising: 

2 g e n e rating at l e ast two independ e ntly adjustabl e interactiv e paramet e rs responsiv e to 

operation of a first touchpad; 
4 gen e rating at l e ast two ind e p e nd e ntly adjustabl e int e ractiv e param e t e rs responsiv e to 

op e ration of a s e cond touchpad; and 
6 controlling said positioning of said first and s e cond cursors bas e d upon int e ractiv e 

param e t e rs g e n e rat e d by on e of said first and s e cond touchpads. 



34t- — A comput e r m e dia product impl e m e nting a m e thod for facilitating comput e r 
2 e diting of an e l e ctronic docum e nt comprising a plurality of obj e cts that ar e graphically 

rend e r e d in a layout comprising a plurality of uniqu e locations, said comput e r m e dia product 
4 comprising comput e r programmabl e cod e impl e m e nting: 

positioning a first cursor at a first location within said e l e ctronic document; 

6 positioning a s e cond cursor at a s e cond location within said e lectronic docum e nt, 

wh e r e in said first and s e cond cursors ar e ind e p e nd e ntly displayabl e and ind e p e nd e ntly 
8 positionabl e at any of said plurality of uniqu e locations of said e l e ctronic docum e nt; 

d e fining a s e l e ction string bas e d upon a s e l e ct e d contiguous arrang e m e nt of at l e ast 

10 on e obj e ct of said plurality of obj e cts; and 

transf e rring said sel e ction string to a location within said e l e ctronic docum e nt as 

12 d e t e rmin e d by said s e cond location of said s e cond cursor. 



2 



3i Th e comput e r m e dia product according to claim 31, wh e r e in said comput e r 

programmabl e cod e furth e r impl e m e nts: 
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rep e at e dly repositioning said s e cond cursor at n e w locations within said e l e ctronic 

4 docum e nt; and 

transferring said s e lection string to e ach of a plurality of now locations within said 

6 e l e ctronic document as d e t e rmin e d by said r e p e at e dly r e position e d s e cond cursor. 

3£ Th e comput e r m e dia product according to claim 31, wh e r e in said first and 

2 s e cond cursors are simultan e ously display e d and positionable in a common graphical user 
interfac e window. 

34: Th e comput e r m e dia product according to claim 31, wh e r e in said computer 

2 programmabl e code furth e r impl e m e nts: 

proc e ssing int e ractiv e param e t e rs r e sponsiv e to op e ration of a hand operat e d us e r 

4 interfac e d e vic e ; and 

s e l e ctiv e ly controlling said positioning of only on e of said first and second cursors at 

6 any giv e n tim e bas e d upon said g e n e rat e d interactiv e param e t e rs. 

35: Th e comput e r m e dia product according to claim 31, wh e rein said comput e r 

2 programmabl e cod e furth e r impl e m e nts: 

proc e ssing at l e ast two diff e r e nt, interactiv e param e t e rs responsiv e to op e ration of a 

4 hand - op e rat e d us e r int e rfac e d e vic e ; and 

controlling said positioning of said first and s e cond cursors bas e d upon said g e nerat e d 

6 at l e ast two int e ractiv e param e t e rs. 

3& Th e comput e r m e dia product according to claim 31, wh e r e in said comput e r 

2 programmabl e cod e furth e r impl e m e nts: 

proc e ssing at l e ast four ind e p e nd e ntly adjustabl e int e ractiv e param e t e rs r e sponsiv e to 

4 op e ration of a touchpad; and 

controlling said positioning of said first and second cursors bas e d upon at l e ast two of 

6 th e g e n e rat e d at l e ast four int e ractiv e param e t e rs. 
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Th e comput e r m e dia product according to claim 31, wh e r e in said computer 

programmabl e cod e furth e r impl e m e nts: 

proc e ssing at l e ast two ind e p e nd e ntly adjustabl e int e ractiv e param e t e rs responsive to 

op e ration of a first trackball; 

g e n e rating at l e ast two ind e p e nd e ntly adjustabl e int e ractiv e param e ters r e sponsive to 

op e ration of a s e cond trackball; and 

controlling said positioning of said first and s e cond cursors bas e d upon int e ractiv e 

param e t e rs g e n e rat e d by on e of said first and s e cond trackballs. 

3& Th e comput e r m e dia product according to claim 31, wher e in said comput e r 

programmabl e cod e furth e r impl e m e nts: 

proc e ssing at l e ast two ind e p e nd e ntly adjustabl e int e ractiv e param e t e rs r e sponsiv e to 

op e ration of a first touchpad; 

proc e ssing at l e ast two ind e p e nd e ntly adjustabl e int e ractiv e param e t e rs r e sponsiv e to 

op e ration of a s e cond touchpad; and 

controlling said positioning of said first and s e cond cursors based upon int e ractiv e 

param e t e rs g e n e rat e d by on e of said first and s e cond touchpads. 

39-. A m e thod for facilitating comput e r editing of an e l e ctronic docum e nt, said 

e l e ctronic docum e nt comprising a plurality of obj e cts that ar e graphically r e nd e r e d in a 
layout comprising a plurality of uniqu e locations, said method comprising: 

positioning a first cursor at a first location within said el e ctronic docum e nt; 

positioning a second cursor at a second location within said e l e ctronic docum e nt, 

wh e r e in said first and s e cond cursors ar e ind e p e nd e ntly displayabl e and indep e nd e ntly 
positionabl e at any of said plurality of uniqu e locations of said electronic docum e nt; and 

d e fining a s e l e ction string bas e d upon a s e l e ct e d contiguous arrang e ment of at l e ast 

on e obj e ct of said plurality of obj e cts, wh e rein said s e l e ction string is capabl e of b e ing 
transf e rr e d to a location within said e l e ctronic docum e nt as d e t e rmined by said s e cond 
location of said s e cond cursor. 
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Ab s tract of th e Di s closure 

Two independ e ntly adjustabl e and positionabl e cursors ar e e mploy e d in a 

visual int e rfac e for e diting an e l e ctronic docum e nt. Th e docum e nt may compris e t e xt 
5 symbols, t e xt obj e cts, and graphics obj e cts, among oth e rs. If both cursors ar e locat e d in 

clos e proximity within the docum e nt, th e n th e s e cursors may b e simultan e ously display e d in 
a singl e window. Oth e rwis e , two ar e as of th e docum e nt, e ach comprising on e of th e cursors, 
may b e simultan e ously display e d using a s e parat e window for e ach cursor, or alt e rnat e ly 
s e l e ct e d and display e d in a singl e window. Copy or cut op e rations may b e mad e with on e 
10 cursor, whil e past e op e rations may b e rep e at e d using the s e cond cursor. Cursor locations 

may b e l e ft unchang e d, or th e y may b e mov e d b e tw ee n or within e diting op e rations. Th e two 
cursors may b e fr ee ly controll e d by an e nhanced pointing d e vic e , such as a mous e with an 
add e d touchpad or trackball, or s e qu e ntially s e l e ct e d and controll e d by a conventional mous e . 



15 Abstract of the Disclosure 

A freely-rotating trackball for simultaneously detecting one, tvvo, or three 
independent directions of its non-rotational displacement and as many as three 
independent directions (roll, pitch, and yaw) of its rotation. In various 

20 implementations, non-rotational displacement of the trackball may be measured or 

interpreted as a widely-varying user interface parameter or as a discrete "click" event 
Signal processing may be used to derive three independent rotation components (roll, 
pitch, and yaw) from more primitive sensor measurements of the trackball. The 
invention provides for trackball displacement and rotation to be sensed by a variety of 

25 sensing techniques including optical, magnetic, electromagnetic, capacitive, resistive, 
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acoustic, resonance, and polarization sensor. The system may be used to provide an 
extended number of simultaneously interactive user interface parameters, and may 
itself be incorporated into larger user interface structures, such as a mouse body. 
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